Application software typically performs tasks by calling shared library functions provided by an operating system or other operating environment through one or more application programming interfaces (APIs). Similarly, malicious software (“malware”) may call system APIs to cause crashes, copy protected data, or perform other malicious activities. Thus, security software may monitor system API calls to detect malware, mitigate damage caused by malware, and perform other security tasks. Security software may monitor API calls by installing one or more inline hooks on the system API. An inline hook is a segment of code that replaces a part of a system API function—typically the first few instructions of the system API function—causing the inline hook to be executed whenever the system API function is invoked. The inline hook may, in turn, redirect program execution to security software that may perform validation checks, remove malware, or perform other security checks. Typical inline hooks are installed within the virtual address space of an application process and thus may be easily overwritten or otherwise circumvented by malware. For example, hook-skipping attacks may simply jump to a location within the system API function that is located past the inline hook.
Typical computer processors include hardware support for virtualization features. Software virtualization includes transparently executing one or more guest operating systems from within a host operating system or VMM. Hardware virtualization features may include an extended privilege model, hardware-assisted support for virtual memory addressing, support for extended memory permissions, and other virtualization features.